System, method, and computer program product for real time transparency-based compositing

ABSTRACT

A system, method, and computer program product for compositing rendered image data in near real time. The input pixel streams that constitute the rendered image data can be video streams, for example. Each input pixel stream can originate from its own graphics processing unit. Each pixel includes a set of color coordinates, such as red, green, and blue (RGB) coordinates, plus an alpha value. Compositing is performed by an image combiner implemented in either hardware or software. The image combiner accepts two or more input pixel streams, and performs compositing on corresponding pixels from each input pixel stream. The compositing process takes into account the color coordinates of each corresponding pixel as well as the alpha values. The compositing process also uses depth information that defines whether a given pixel is in the foreground or in the background relative to another corresponding pixel. The result of the compositing process is a resultant pixel stream based on corresponding pixels of each input pixel stream.

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is a continuation-in-part of U.S. patentapplication Ser. No. 09/888,438, filed Jun. 26, 2001, which claimspriority to U.S. Provisional Application No. 60/219,006, filed Jul. 18,2000. U.S. patent application Ser. Nos. 09/888,438 and 60/219,006 areboth incorporated herein by reference in their entireties.

STATEMENT REGARDING FEDERALLY-SPONSORED RESEARCH AND DEVELOPMENT

[0002] Not applicable.

REFERENCE TO MICROFICHE APPENDIX/SEQUENCE LISTING/TABLE/COMPUTER PROGRAMLISTING APPENDIX (submitted on a compact disc and anincorporation-by-reference of the material on the compact disc)

[0003] Not applicable.

BACKGROUND OF THE INVENTION

[0004] 1. Field of the Invention

[0005] The invention described herein relates to computer graphics, andmore particularly to compositing of images.

[0006] 2. Background Art

[0007] A common problem in computer graphics is the efficientcompositing of two or more rendered images to produce a single image.The compositing process typically involves the combination of theimages, pixel by pixel, and takes into account the respective colorcoordinates of each pixel. The process can also take into accountopacity, intensity, and the relative distances of the images from aviewer.

[0008] One way in which compositing has been accomplished in the past isthrough software-based readback from frame buffers. In sucharrangements, two or more graphics processors each send frames ofrendered graphics data to their respective frame buffers. The contentsof each frame buffer are then read back, into a compositor module. Thecompositor module can be a compositing frame buffer or a graphics host.At the compositor module, software-based compositing is performed oncorresponding pixels, one pixel from each frame buffer. A pixel from oneframe buffer is composited with a pixel from the second frame buffer.This continues until all appropriate pixels in each frame buffer havebeen composited. A final output, comprising the resultant compositedpixels, is then available from the compositor module.

[0009] In some applications such a system can be adequate. Inapplications requiring faster compositing, however, such a system maynot be fast enough. In video applications, for example, where the finaloutput must be produced in real time, such compositing entailsunacceptable delay.

[0010] Hence there is a need for a system and method for fastcompositing, where the compositing can take place at near real timerates.

BRIEF SUMMARY OF THE INVENTION

[0011] The invention described herein is a system, method, and computerprogram product for compositing rendered image data in real time or nearreal time. The input pixel streams that constitute the rendered imagedata can be video streams, for example. Each input pixel stream canoriginate from its own graphics processing unit. Each pixel includes aset of color coordinates, such as red, green, and blue (RGB)coordinates, and can also include an alpha value. Compositing isperformed by an image combiner that can be implemented in eitherhardware or software. The image combiner accepts two or more input pixelstreams, and performs compositing on corresponding pixels from eachinput pixel stream. The compositing process takes into account the colorcoordinates of each corresponding pixel as well as the alpha values orintensity values. The compositing process also uses depth informationthat defines whether a given pixel is in the foreground or in thebackground relative to another corresponding pixel. At the pixel level,the result of the compositing process is a resultant pixel based on thecorresponding pixels of each input pixel stream.

[0012] The foregoing and other features and advantages of the inventionwill be apparent from the following, more particular description of apreferred embodiment of the invention, as illustrated in theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS/FIGURES

[0013]FIG. 1 is a block diagram illustrating a system for real time ornear real time combination of graphics images, according to anembodiment of the invention.

[0014]FIG. 2 is a block diagram illustrating an image combiner,according to an embodiment of the invention.

[0015]FIG. 3 is a flowchart illustrating a method for real time or nearreal time compositing of graphics images, according to an embodiment ofthe invention.

[0016]FIG. 4 is a flowchart illustrating the compositing process,according to an embodiment of the invention.

[0017]FIG. 5 is a flowchart illustrating a method for real time or nearreal time compositing of graphics images wherein inputs representadjacent volumes of a three-dimensional scene, according to anembodiment of the invention.

DETAILED DESCRIPTION OF THE INVENTION I. Overview

[0018] The invention described herein is a system, method, and computerprogram product for the real time compositing of two or more input pixelstreams. The input pixel streams can be video streams, for example. Eachinput pixel stream can originate from its own graphics processing unit.Each pixel includes a set of color coordinates, such as red, green, andblue (RGB) coordinates, plus an alpha value that defines the opacity ofthe pixel. Compositing is performed by an image combiner that can beimplemented in either hardware or software. The image combiner acceptsthe input pixel streams, and performs compositing on correspondingpixels from each input pixel stream. The compositing process takes intoaccount the color coordinates of each corresponding pixel as well as thealpha values or the intensity values. The compositing process also usesdepth information that defines whether a given pixel stream is in theforeground or in the background relative to another corresponding pixelstream. At the pixel level, the result of the compositing process is aresultant pixel stream based on the corresponding pixels of each inputpixel stream.

II. System

[0019] An exemplary context for the invention is illustrated in FIG. 1.Two graphics processors are shown, processors 105 and 110. Graphicsprocessor 105 produces a frame of image data; likewise, graphicsprocessor 110 produces a frame of image data. The image data of graphicsprocessor 105 is then sent, as input 121, to image combiner 130.Similarly, the rendered image data produced by graphics processor 110 issent as input 122 to combiner 130. Combiner 130 also receives depthinformation 132. Depth information 132 indicates the depth order of theinputs 121 and 122. The inputs can be understood as images to becombined; depth information 132 indicates which is in the foreground andwhich is in the background, relative to a viewer. Depth information 132generally does not change in the context of a given frame.

[0020] Combiner 130 performs compositing on corresponding pixels ofinputs 121 and 122 based on the color coordinates of the correspondingpixels, the alpha values of the corresponding pixels, and the depthinformation 132. The compositing process is described in greater detailin section III below. A resultant pixel stream 135 is then produced bycombiner 130. In an embodiment of the invention, the pixels of pixelstream 135 that constitute a frame are stored in a frame buffer 140. Inthis embodiment, the output of system 100 is a frame of image data,output 145.

[0021] Note that while the embodiment of FIG. 1 shows two graphicsprocessors and two associated inputs, other embodiments of the inventioncould feature more than two inputs. In such a case, the compositorperforms compositing on all the inputs, taking into account the relativedepth information of all inputs. Also, inputs 121 and 122 of FIG. 1 arereceived by combiner 130 from respective graphics processors 105 and110. In an alternative context, however, inputs to combiner 130 caninclude the outputs of other image combiners.

[0022] Image combiner 130 is illustrated in greater detail in FIG. 2.Combiner 130 includes a depth determination module 205. Depthdetermination module 205 receives depth information 132. As describedabove, depth information 132 indicates the depth order of the imagesrepresented by the input pixel streams. Depth determination module 205coverts this information to a format usable for blending purposes.Output 210 of depth determination module 205 therefore conveys whichinput pixel stream is “over” another.

[0023] Output 210 is sent to one or more blending modules, shown in FIG.2 as blending modules 215 through 225. In an embodiment of theinvention, each blending module is associated with a specific colorcoordinate. In the embodiment of FIG. 3, blending modules 215, 220, and225 are associated with red, green, and blue coordinates, respectively.Each blending module performs blending of a color from correspondingpixels from respective input pixel streams. Hence, blending module 215blends red coordinates R₁ and R₂ from corresponding pixels. The alphavalues of the corresponding pixels, α₁, and α₂, are also input toblending module 215, along with output 210 of depth determination module205. Inputs to blending modules 220 and 225 are analogous.

[0024] The invention can implement any of several well known blendingoperations. In a preferred embodiment, blending is performed in depthorder, taking into account the opacity of input pixels. Here, colors areblended linearly according to alpha values. In one embodiment, a blendedred coordinate, for example, has the value α₁R₁+(1−α₁)α₂R₂, assumingthat input 1 is over input 2. The other color coordinates are blendedanalogously. Alternatively, if compositing is based on maximumintensity, the value of the resultant red coordinate is the maximum ofthe red coordinates of the corresponding pixels. The other colorcoordinates would be blended analogously. Compositing is described morefully in Computer Graphics, Principle and Practice, Foley et al.,Addison-Wesley, 1990, pp.835-843 (included herein by reference in itsentirety).

[0025] The composited color coordinates from blending modules 215through 225 are then sent to an output module 230 for formatting asresultant pixel 235.

[0026] While the embodiment of FIG. 2 shows the blending of colorcoordinates in parallel, alternative embodiments can perform blending ofcolor coordinates in serial using a single blending module.

III. Method

[0027] The method of the invention is illustrated in FIG. 3. The methodbegins at step 305. In steps 310 and 315, two inputs, shown here asinputs 1 and 2 are received at a compositor. In step 320, depthinformation is received by the compositor. In step 325, the compositingof the two inputs is performed in depth order, so as to take intoaccount the depth information received in step 320. The processconcludes at step 335.

[0028] Compositing step 325 is illustrated in greater detail in FIG. 4,according to an embodiment of the invention. The compositing processbegins at step 405. In step 410, the depth order of the input pixelstreams is determined for a frame. This determination is made based onthe received depth information. In step 415, the first color coordinatesfrom corresponding pixels are blended, based on the depth order and thepixels' alpha values. Similarly, in steps 420 and 425, the second andthird color coordinates are blended. In step 430, a resultant pixel isoutput. In step 435, a determination is made as to whether additionalpixels are to be blended for the current frame. If so, the processrepeats from step 415 with a new set of corresponding pixels. Otherwise,the process ends at step 440.

[0029] As mentioned above, the process of compositing pixels based ontransparency is known to persons of ordinary skill in the art, and isdocumented in Computer Graphics, Principle and Practice, Foley et al.,supra. Moreover, while the embodiment of FIG. 4 shows the blending ofcolor coordinates in parallel, alternative embodiments can performblending of color coordinates in serial.

[0030] A particular embodiment of the method of the invention is shownin FIG. 5. The method begins with step 505. In steps 510 and 515, inputs1 and 2 are received respectively. In this embodiment however, eachinput represents a sub-volume of three dimensional space from a scenethat has been rendered. The sub-volumes represented by inputs 1 and 2can therefore be thought of as a back “slab” and a front slab,respectively, where the terms front and back refer to the relativepositions of each volume from the current frame perspective of theviewer. In step 520, the depth information is received, as before. Instep 525, composition is performed on the slabs in depth order, and theresultant pixel stream is output. The process concludes at step 535.

[0031] Note that while the embodiments of FIGS. 3 through 5 shows twographics processors and two associated inputs, other embodiments of theinvention could feature more than two inputs. In such a case, thecompositor performs compositing on all the inputs, taking into accountdepth information relating to the inputs.

IV. Computing Environment

[0032] Referring to FIG. 2, compositor 130 may be implemented usinghardware, software or a combination thereof. In particular, compositor130 may be implemented using a computer program, and execute on acomputer system or other processing system. An example of such acomputer system 600 is shown in FIG. 6. The computer system 600 includesone or more processors, such as processor 604. The processor 604 isconnected to a communication infrastructure 606 (e.g., a bus ornetwork). Various software embodiments can be described in terms of thisexemplary computer system. After reading this description, it willbecome apparent to a person skilled in the relevant art how to implementthe invention using other computer systems and/or computerarchitectures.

[0033] Computer system 600 also includes a main memory 608, preferablyrandom access memory (RAM), and may also include a secondary memory 610.The secondary memory 610 may include, for example, a hard disk drive 612and/or a removable storage drive 614, representing a magnetic mediumdrive, an optical disk drive, etc. The removable storage drive 614 readsfrom and/or writes to a removable storage unit 618 in a well knownmanner. Removable storage unit 618 represents a magnetic medium, opticaldisk, etc. As will be appreciated, the removable storage unit 618includes a computer usable storage medium having stored therein computersoftware and/or data.

[0034] Secondary memory 610 can also include other similar means forallowing computer programs or input data to be loaded into computersystem 600. Such means may include, for example, a removable storageunit 622 and an interface 620. Examples of such means also include aprogram cartridge and cartridge interface (such as that found in videogame devices), a removable memory chip (such as an EPROM, or PROM) andassociated socket, and other removable storage units 622 and interfaces620 which allow software and data to be transferred from the removablestorage unit 622 to computer system 600.

[0035] Computer system 600 may also include a communications interface624. Communications interface 624 allows software and data to betransferred between computer system 600 and external devices. Examplesof communications interface 624 may include a modem, a network interface(such as an Ethernet card), a communications port, a PCMCIA slot andcard, etc. Software and data transferred via communications interface624 are in the form of signals 628 which may be electronic,electromagnetic, optical or other signals capable of being received bycommunications interface 624. These signals 628 are provided tocommunications interface 624 via a communications path (i.e., channel)626. This channel 626 carries signals 628 into and out of computersystem 600, and may be implemented using wire or cable, fiber optics, aphone line, a cellular phone link, an RF link or other communicationschannels. In an embodiment of the invention, signals 628 can compriseimage data (such as inputs 221 and 222) and depth information (such asdepth information 232).

[0036] In this document, the terms “computer program medium” and“computer usable medium” are used to generally refer to media such asremovable storage drive 614, a hard disk installed in hard disk drive612, and signals 628. These computer program products are means forproviding software to computer system 600. The invention is directed inpart to such computer program products.

[0037] Computer programs (also called computer control logic) are storedin main memory 608 and/or secondary memory 610. Computer programs mayalso be received via communications interface 624. Such computerprograms, when executed, enable the computer system 600 to perform thefeatures of the present invention as discussed herein. In particular,the computer programs, when executed, enable the processor 604 toperform the features of the present invention. Accordingly, suchcomputer programs represent controllers of the computer system 600.

V. Conclusion

[0038] While various embodiments of the present invention have beendescribed above, it should be understood that they have been presentedby way of example, and not limitation. It will be apparent to personsskilled in the relevant art that various changes in detail can be madetherein without departing from the spirit and scope of the invention.Thus the present invention should not be limited by any of theabove-described exemplary embodiments, but should be defined only inaccordance with the following claims and their equivalents.

What is claimed is:
 1. A method of combining a plurality of input pixelstreams to form a resultant pixel stream, comprising the steps of: a)receiving the plurality of input pixel streams; b) receiving depthinformation relating to the relative depth of the input pixel streams;c) compositing, in depth order, corresponding pixels from each inputpixel stream to form the resultant pixel stream in approximately realtime.
 2. The method of claim 1, wherein each pixel comprises colorcoordinates and an alpha value.
 3. The method of claim 1, wherein stepc) comprises the steps of: i) blending the corresponding pixelsaccording to the alpha values of the corresponding pixels and the depthinformation; ii) outputting the resultant pixel; and iii) if the inputpixel streams contain additional corresponding pixels, repeating steps(i) and (ii) for the additional corresponding pixels.
 4. The method ofclaim 1, wherein the input pixel streams and resultant pixel stream arevideo streams.
 5. The method of claim 1, wherein the input pixel streamseach represent renderings of adjacent sub-volumes, such that theresultant pixel stream represents a rendering of the adjacentsub-volumes viewed collectively.
 6. The method of claim 1, wherein thedepth information can vary for each frame.
 7. An image combiner forcombining a plurality of input pixel streams to form a single resultantpixel stream, comprising: a depth determination module for convertingdepth information to an indication as to depth order of the input pixelstreams; and one or more blending modules that perform a blendingoperation on color coordinates of corresponding input pixels, on thebasis of said depth order and alpha values associated with saidcorresponding input pixels.
 8. The system of claim 7, wherein the inputpixel streams comprise image data output from a graphics processor. 9.The system of claim 7, wherein the input pixel streams comprise imagedata output from another image combiner.
 10. A computer program productcomprising a computer usable medium having computer readable programcode means embodied in said medium for causing a program to execute on acomputer that combines a plurality of input pixel streams to form asingle resultant pixel stream, said computer readable program code meanscomprising: a first computer program code means for causing the computerto receive the plurality of input pixel streams; a second computerprogram code means for causing the computer to receive depth informationrelating to the relative depth of the input pixel streams; and a thirdcomputer program code means for causing the computer to composite, indepth order, corresponding pixels from each input pixel stream to formthe resultant pixel stream in approximately real time.
 11. The computerprogram product of claim 10, wherein each pixel comprises colorcoordinates and an alpha value.
 12. The computer program product ofclaim 10, wherein said third computer program code means comprises: i)computer program code means for combining the corresponding pixelsaccording to the alpha values of the corresponding pixels and the depthinformation; ii) computer program code means for outputting theresultant pixel; and iii) computer program code means for repeatingexecution of code means (i) and (ii) for additional correspondingpixels, if the input pixel streams contain additional correspondingpixels.
 13. The computer program product of claim 10, wherein the inputpixel streams and resultant pixel stream are video streams.
 14. Thecomputer program product of claim 10, wherein the input pixel streamseach represent renderings of adjacent sub-volumes, such that theresultant pixel stream represents a rendering of the adjacentsub-volumes viewed collectively.
 15. The computer program product ofclaim 10, wherein the depth information can vary for each frame.